fix(mysql): make PITR disk-pressure binlog cleanup functional and prefix-safe#3088
Draft
weicao wants to merge 6 commits into
Draft
fix(mysql): make PITR disk-pressure binlog cleanup functional and prefix-safe#3088weicao wants to merge 6 commits into
weicao wants to merge 6 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3088 +/- ##
=====================================
Coverage 0.00% 0.00%
=====================================
Files 147 147
Lines 23282 23282
=====================================
Misses 23282 23282 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…fix-safe Replica discovery relied on KB_ITS_*_HOSTNAME env that never existed in DataProtection job pods on KB 1.x, so the >=80% disk cleanup (default PURGE_BINLOG=on) silently never purged anything. Discover replicas from the target primary via SHOW REPLICAS / SHOW SLAVE HOSTS instead. Also make the master purge prefix-safe: PURGE BINARY LOGS TO is prefix deletion, so scan stops at the first file that must be kept (not yet synced+uploaded, or one of the newest 5) and a single PURGE TO that file is issued - the old per-file loop could delete an unarchived binlog sitting before a qualifying one and tear the archived sequence. Membership tests are now exact-match instead of substring grep. Fixes #3087 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The disk-pressure binlog purge only triggers at >=80% usage, which cannot be safely reached on a large shared host volume in tests (would need a ~177GB fill). Make the threshold overridable via PURGE_BINLOG_DISK_THRESHOLD (default 80, production behaviour unchanged) so a test can set it below the current usage and exercise the real purge path without filling the disk. Refs #3087 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
get_synced_binlogs matched replica status against a reconstructed
${DP_TARGET_POD_NAME}-bin, which fails when the target env drifts from
the actual binlog-writing pod (e.g. after a switchover) - returning
'No synced binlog files found' even when replicas are synced. Match
against LOG_PREFIX (derived from the real log_bin_basename) instead.
Add DEBUG lines printing DP_TARGET_POD_NAME/LOG_PREFIX/replica hosts and
per-replica resolved current_file, so the cleanup-decision path is
observable in the archive-binlog log.
Refs #3087
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stderr
get_replica_hosts assumed replicas register report_host, but the semisync
cmpd start command omits --report-host (only the MGR cmpd and
mysql-entrypoint.sh set it). With report_host empty, SHOW REPLICAS leaves the
Host column blank and `awk '{print $2}'` slides onto the Port, returning
"3306" as the replica host (observed: 'Processing replica host: 3306',
'Can't connect to MySQL server on 3306:3306'). Discover connected replicas
from the primary's own Binlog Dump threads instead
(information_schema.PROCESSLIST), which carry the replica ip:port regardless
of report_host; strip the ephemeral :port. Fall back to SHOW REPLICAS / SHOW
SLAVE HOSTS with TAB-split awk so an empty Host stays empty instead of
becoming the Port.
Also move the two DEBUG lines in get_synced_binlogs to stderr: that
function's stdout is captured by `synced_binlogs=$(get_synced_binlogs)`, so
echoing on stdout both hid the lines from the pod log and corrupted the
return value (making the empty result look non-empty and bypassing the
fail-closed 'No synced binlog files found' guard).
Split the `local synced_binlogs=$(...)` declaration from its assignment so
the `$? -ne 0` check reflects get_synced_binlogs' exit code rather than the
always-zero exit of `local`, keeping the fail-closed guard closed.
Refs #3087
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two data-safety defects found in a deep review of the PITR binlog cleanup. 1. The fail-closed guard `synced_binlogs=$(get_synced_binlogs); if [ $? -ne 0 ]` is unreachable under the actionset-wide `set -e`: a separated `var=$(fn)` assignment where fn returns 1 aborts the whole script before the check, so a replica-parse failure crash-loops the archiver instead of logging and skipping the purge. (Splitting the earlier `local var=$(...)` to fix `$?` masking removed the mask that `set -e` was relying on.) Use `if ! synced_binlogs=$(get_synced_binlogs) || [ -z ... ]` so the failure is caught without tripping set -e. 2. `min_synced_file` was a leaked global, never reset. On a 2nd+ cleanup call in the long-lived archiver, if replica discovery returns empty the loop is skipped and the stale previous value passes the guard, purging binlogs on an unverified synced position (potential data loss). Declare it `local` with an empty reset at function entry. Both verified: bash confirms the separated assignment aborts under set -e while `if !` does not; helm render + bash -n pass.
weicao
force-pushed
the
helios/mysql-pitr-binlog-cleanup
branch
from
July 13, 2026 12:54
52d78d4 to
7dcbd13
Compare
Contributor
Author
|
Absorbed the remaining follow-up from PR #3157 at head 774786e. The existing main-based PR already contained the set-e and stale-state guards; this update additionally matches uploaded binlogs by the live LOG_PREFIX instead of stale DP_TARGET_POD_NAME. bash -n, helm lint, and git diff --check pass. PR #3157 is superseded by this PR. |
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3087.
Problem
The PITR disk-pressure cleanup path could misclassify binlog state or continue after command failures:
LOG_PREFIX;Changes
SHOW REPLICAS, with the MySQL 5.7 fallback;PURGE BINARY LOGS TOonly for the safe prefix;LOG_PREFIX;Validation
bash -non the touched script: PASShelm lint addons/mysql: PASSgit diff --check: PASSEvidence Boundary
These are source, syntax, render, and CI gates. Runtime PITR/disk-pressure behavior remains N=0 and belongs to the test owner.